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ABSTRACT 

To determine whether placement of items on a teacher 
rating scale affects the factor structure underlying the scale and to 
determine whether changing the item format alters the ratings given, 
a 12-item, high-inference student rating scale was developed 
containing two global items pertaining to overall teacher 
effectiveness and 10 evaluative items pertaining to more specific 
teacher characteristics. Four different formats were constructed and 
randomly distributed to 3 63 college students within 27 university 
classes. The results of a principal components analysis suggest that 
the factor structure underlying the rating instrument is stable and 
not affected by changes in the placement of items. The nonsignificant 
results from a multivariate analysis of variance (MANOVA) provide 
additional support for the stability of the ratings. The MANOVA 
results in particular suggest that student ratings do not increase or 
decrease as a result of changes in item placement. The results of 
both analyses detract from the view that the halo effect may be 
influenced by the positioning of items. There is thus little reason 
to believe that preliminary items may set the stage for responses on 
following items. (RC) 
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The Influence of Format Change on the 
Halo Effect of Student Ratings 

Robin D« Proman 
University of Connecticut 

Introduction and Review 

One of the chronic problems of student ratings has been estab- 
lishing content validity. Studies seeking to establish the technical 
adequacy of student ratings have generally supported the reliability 
of the measures, but validity remains controversial. It is not clear 
whether student ratings of teacher performance are actually reflections 
of teacher characteristics and teaching ability, or if they are in some 
part a reflection of student characteristics* 

The halo effect has been claimed to cloud the interpretation of 
rating outcomes. The halo effect, in brief, is the tendency for raters 
to evaluate conceptually different characteristics as thou^ they all 
represented the same thing* It is the tendency for prior information, 
or one impression or characteristic, to influence all other impressions* 
In other words, if an instructor receives a hi^ rating on one charac- 
teristic, it is likely that he will receive a hi^ rating on other 
characteristics* 

Person-perception research has built a considerable argument for 
the existence of implicit personality theories , one possible cause or 
explanation for the halo effect* In a review of the research on the 
effects of implicit personality theory, Schneider (l973) suggests that 
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rat: nstnimentB are more affected by the preconceived personality 
theory of the rater than by the characteristics of the person being 
rated. Evidence is cited in the review to illustrate that similar trait 
structures emerge when different rating instruments are factor analyzed. 
These structures are virtually unaffected by the stimulus object, the 
familiarity of the rater with the ratee, differences in samples, or 
differences in situations (see, for example, Magoon and Price, 1972) • 

Within the literature pertaining specifically to teacher evaluation 
instruments, consideration has been given to the effect of implicit per- 
sonality theory. Johnson, Rhodes, and Rumeiy (l973) suggest that teacher 
rating instruments do not necessarily furnish information about teacher 
performance, but rather, provide information about the acceptability of 
teacher characteristics and performance. Thus, teacher rating scales 
would appear to provide more information about student perceptions and 
implicit personality theories than about actual teacher behaviors. This 
notion is supported by the results of a study by riagoon and Price (1972). 
When comparing the results of principal component analyses of 21 sets of 
rating data between and within classrooms, the authors found that the 
major dimensions ^Aiich emerged were very similar in structure. These 
researchers concluded that these striking similarities provided evidence 
that ratings reveal more about student preconceptions about the relation- 
ships between teacher traits thaui about the quality of instruction. In 
a later investigation of this possibility, Bausell and Magoon (1972) 
found a correlation of .67 between evaluations of teaching performance 
made on the first day of class and evaluationB made at the end of the 
semester. In addition, the authors found that the changes in ratings 



that did occur were predictable and related to changes in the grade a 
student expected to receive. If implicit j>ersonality theories are oper- 
ating, one wonders whether stability estimates (of reliability) indicate 
consistency in teacher performance or consistency in students* beliefs. 

Several studies have been conducted to determine what variables ao 
or do not affect the bias in ratings which result from a rater's implicit 
personality theory. Schneider (1973) reported that occupation of the 
rater, caffeine arousal, axid the perceived importance of the evaluation 
were found to influence the factor structures which emerged from analyses 
of rating instruments. Slater and Owen (1974) found that the major area 
of study of students affected the factor structure which emerged when . 
students' ratings of hypothetical "ideal professor" were analyzed. The 
differences in the factors due to a student's major suggest that one's 
field of study may be a relevant predictor of the bias or implicit per- 
sonality theory a student has. 

Dixon and Dixon (1975) found this bias to be affected by the over- 
all perception a student has of a teacher's competence. A comparison 
of teachers rated as "exceptionally good" versus teachers rated as 
"exceptionally poor" yielded two different sets of factor structures. 
The differences in the factor structures imply that students have dif- 
ferent implicit personality theories with respect to good teachers and 
poor teachers. 

A study by Pollman, Lucoff, Small, and Power (1974) reports quanti- 
tative differences in ratings given to teachers as a result of format 
changes in scaling, coding, and response alternatives. There were, 
however, no analyses reported which would determine if there were changes 
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in the factor structures as a result of format changes. Thus the 
Follman et al« study shows evidence of changes in the mean ratings 
awarded based upon format changes, but does not determine if the dimen- 
sions underlying student bias change as a result of format changes. 

The purpose of the present investigation was to determine if simple 
format changes would alter the existing halo effect and hence influence 
student ratings of teacher performance. The research was designed to 
find out whether the placement of questions on a rating scale is respon- 
sible for any halo effect. In particular, it was felt that global items 
pertaining to teacher effectiveness mighic provide a certain "tone" or 
"set" for students as they respond to subsequent questions. That is, 
having responded to initial items, the students mi^t simply "follow 
suit" on the rest of the items on a scale. 

The specific objectives of this study were twofold: 

1. To determine whether placement of items on a teacher rating 
scale affects the factor structure underlying the scale; and 

2. to determine whether changing the item format alters the 
ratings given. 

Procedure 

A twelve-item, high-inference student rating scale was developed 
containing two global items pertaining to overall teacher effectiveness 
and ten evaluative items pertaining to more specific teacher character- 
istics (see Appendix A).^ 

Pour different formats were constructed, varying the items as follows 
1. The two global items were at the top of the rating instrument; 

^Por more information concerning the construction, validity, and 
reliability of the rating scale refer to. Student Ratings; What 
is the Prame of Reference? by A. Lolli and S. Owen, a paper 
presented at the Annual Convention of NOME, San Prancisco, 1976. 



2. The two global items were at the bottom of the rating instrument; 
3» The two global items were absent from the rating instrument; 
4* The two global items were the onl^ items present on the form* 
The four formats are shown in Appendices A throu^ D« The four forms 

were ramdomly distributed to 363 students within 27 universitity classes* 

2 

Undergraduate and graduate students from four academic areas were repre- 
sented in the sample* 

Data from all classes were pooled and three principal components 
analyses were conducted to identify the factor sti*uctures underlying the 
ten evaluative items* Data from forms ly 2, and 3 were considered in 
the factor analysis* A one-way MANOVA was applied to the data from 
Forms 1, 2, and 4t considering the two global items as the dependent 
variables* A second one-way MANOVA was performed on forms 1, 2, and 3f 
considering the ten evaluative items as the dependent measures (see 
Figure l)* 

Results 

Table 1 shows the results of the principal components analysis* 
The factor structures across the three forms containing the ten evalua- 
tive items were very similar* The strongest five items loading above 
*40 on Factor I were the same across the three forms, with sli^t changes 
in the order of item importance* The average percent of variance ac- 
counted for by Factor I was *39f the range from *34 to *42* Analyses 
of the data from form 3 yielded only two factors; the data from forms 

2 

The academic aireas represented in the sample were educationi 
psychology, engineering and history* 

5 
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1 and 2 yielded three factors. The two factor solution resulting from 
form 3 was apparently the consequence of collapsing the second and third 
factors which emerged on the other two forms into one factor. The col- 
lapsing of the factors was probably due to the eigenvalue criterion of 
1.0. A third factor, quite similar to the third factor seen in forms 
1 and 2, developed an eigenvalue of .95 and was not retained for rotation. 

The slight changes in the relative importance of items in defining 
factors two and three on forms 1 and 2 can be explained as the result 
of a restriction of range on those items. In particular, items 5 SLnd 7 
had average variances of .37 and .21 respectively across the three forms. 
The average variances of the ei^t other evaluative items across the 
three forms compared in the factor analysis was •76. 



Insert Table 1 about here 

Two MANOVA's were conducted to determine if format changes would 
alter the ratings given. Neither MANOVA showed differences in the mean 
vectors of the dependent variables. The MANOVA analyzing tho mean vectors 
of the ten evaluative items produced an F value of 0.83ff p < .68| with 
degrees of freedom of 20 and 702. The analysis of the forms containing 
only the two global items showed an F value of .73^ p < .34? with degrees 
of freedom of 4 and 710. 

Conclusions 

The results of the principal components analyses suggest that the 
factor structure underlying the rating instrument is stable and not 
affected by changes in the placement of the items. The nonsignificant 
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results from the MANOVA provide additional support for the stability 
of the ratings* The MANOVA results in particular suggest that student 
ratings do not increase or decrease as a result of changes in item 
placement. The results of both analyses detract from the view that the 
halo effect may be influenced by the positioning of items. There is 
thus little reason to believe that preliminary items may "set the stage" 
for responses on following items. 

Althou^ this research has not pinpointed the reasons for the halo 
effect, the negative findings may be viewed as a fortunate outcome. 
Researchers in instrument development may be somewhat more confident 
that general format revisions will not influence the technical adequacy 
of teacher rating scales. 
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ilPPENDIX A (i-'orm l) 
The University of Connecticut Survey of Courses and Teaching 



Department Coui'^e'"^ rnstructor 

PIIiECTIOMS : Student ratings caoi be a powerful type of feedback to the instructor. 

£j:£il^J^lA^<iy.ecjLj^^^^^ ^ )!SH-S.^: ,^01I0T SI CM YOU R^ M^/. 

Your Seme.jter: S ' g '3 ^V'g 'T VoFid? ^^^^^W ^ 

Expected grade in course ' c ' P ^ AS ^ Cumulative Grade Average; 

1.0-1.5 1] 2.5-3.0"* 



Perceat of class attendance: 
50-74 



25-49 
0-24 



Io5-2o0 
2.0-2o5 



3-0-3-5 
3-5-4-0 



Time spent on this course compared to other courses of 
equal credit: considerably more 

somewhat more 
about the same 
somewhat less 
considerably l e s s^ 



How much have you leamed_from this course? 

almoflfir nothing aSli^tle a moderate quit e^ a bit a gr^55? deal 

amount 

How v/ouldjrou rate this instructor in ^neral, all-around t e aching abi 1 it y^ 



pool 



leBs"l^han 
adequate 



avey^ige 



good. 



excellent 



For the following ten questions, rate each item in two ways: 
av give ycur act ual rating of the instructor; then 

b. mark the second scale according to how you would prefer things to bc« Of 
course, if you are satisfied, the second scale will be marked the same as the 
first . 



o^l oo p tJO 



Prom my observations, this instructor: 



B 

H- 

g- 

O 



(003 XB 



1. presents material in a clear & effective manner 

(preferred^ b»v('*«BV>avAooa3o«oe«e«a*(>«ooaflooooo«o«oaoi.oool 

2 % St imUlat OS int crest ■>o<>w(.vow*aacno03«««eo*ewoo«ii»woo«:«bb;)^oo«tiec| 
(preferred^ ob»«eo«»oaoontitioaoo«BB«etioo«oo«oa«ooo«aooOi>a«| 

3« makes work assignments & otudont rcoponsibiliuics clear* • • « o . s ' 

(preferred^ «ae»»«BBeeBee««eoat»jajo«tiajoa«aa«aaobOo«o«oa« 

4« USDS exam items which stress important aspects of the courscc/ 

(preferred ^o«.i««t*««B«o«««eaooaaeeooatie«o««av cQ««oftO aoaob 

5* grades fairly end impartially « - . o . . . « « . . . . • o -7 

6* demonstrates overall organization- c ....ao ' 

(preferred) - a • « » . . . c - a - « a o"" 

?• moots class regularly and on timcc 
(preferred) a .... a 



_L..J 



-4 



— i 



. a • . a • 



laao... .av Aaov* aob.a' 



8. fulfills class objectives and obligations a . 

(preferred) . » a « * o a a * . c ^ 



««««sa..i>aaa.ob 



-1 
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9« is accessible to students both in and out of class 

(preferred) ••co«...a««a«.8...a.o.eaoafta.a»aoac€*joft«aoi»oo 

10. shows an interest in and concern for studcntSv. 



r-1 — r-i 



preferred) . , 



It 



• «i.»**a«eMO| 

o .] 



iiPPENDIX B (Form 2) 
The University of Coiinecticut Survey of Courses and Teaching 



Department _ 



Course # Instructor 



]^^ii2SSS Your instructors will not see these ratings until after grades are 

awarded . PlS5 £S-^. objective as j^i^£afl^ari_d^J?OT .SlffiMfOUR H AJEj 

Your semester: OT-4^r^ipn^3^:k Iq this course in your major fTe'ld?l.:.^J O 

Expected grade in course: i J-^-^^^— ' 

A 3 C D P Pass Cumulative grade average: 



Percent of class attendence: 
90-100% 



75-89 ;i 
50-74 
25-49 T. 
0-24 



2.5-3.0 
3.0-3.5 
3o5-4.0 



1 .0-1 .5 •" 
1 .5-2.0 
2.0-2.5 

Time spent on this course compared to other coursesof 
equal credit: considerably more !~ 

somewhat more 
about the some 
somewhat less 
considera bly less ^ 



uor the following ten questions, rate each ifom in two "ways: 
a. give your actual rating of the instructor; then 
b= mark the second scale according to how you would prefe r things to be. Of 

course, if you»ro satisfied, the 2nd scale will be marked the oamo as the 1st, 



Prom my observations, this instructor... 



1. presents material in a clear & effective manner........ ; 

Vpref erred) , c . . ^ „ , ^ ^ : • 

2« otimulatcs interest c .0.. p 

(preferred) ' ' ' ' ] ! ! 1 ! 1 ] ! [ ] ] ^ [ ! i ] [ ] [ ^ ] ] ] ^ ; ] ^ ] ] ] ] 



P 

OB 
<o 

OUi 
4c+ 



. ;j : CO , 

00 o I 



3 I Oil 
O I c4. 

HO 

C • C 



c 



CQc4- 



3.. makes work assignments & student responsibilitieG clear......;" 

(pref erred) " j- 



4.UCOS exam items which stress importcUit aspects of the course, 
(preferred) 



5. grMes fairly and impartially , „ ^ 

(preferred).,,,,.. .i 

6. demonstrates overall organization 

(preferred) J 

7. meets class regularly and on time....,.,, 

(preferred) , . . , 

8. fulfiUa class objectives and obligations. 

(preferred),.,,,.,,, 



o 0 a 0 1 

O O 0 tl « ti It 



1; ••>oooaoo9tio, 



9. is accessible to students both in and out ef class 

(preferred) ^ 

10. Shows an interest in and concdm for students. 



c 11 0 o w a ( 



^ 

How auch^tovo^ycu ~iea'i^_Trom'' thrr"MliTs'eT 



almost nothing 



a little 



quite a bit 



CI3 
great deal 



ERIC 



now weul^you rate thisjnstructor in^eneral, overaJJ teaching .^ility^ 
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APPENDIX C (Form 3) 
The University of Connecticut Sxxrvey of Courscc and Teaching 

Department ^ Course Instructor 

£Lr2°Ji2P.°-* You- instructors will not see these ratings until after grades are 
-SHa£i®Ai-jyi?AsoJbe.il£^^ .9B}j.J3^LI^^^yL Slffl'?, YOiffi NME; 

Your semester;!- i_i I Lt^ l I ^ I -r . .— 'i i — > 
1 2 3 4 5 6 7 8 Grad . Is this course in your major fiGld?L-J \~~\ 

E3q?octod grade in course ; rJ^ J~r I I 1 Cvvnulativo grgdo average: 

A B C D P Pass ^ ^^f-j 2.5.3,0 | [ 

^ -5-2.0^2 3«0"3.5 I 

2.o-2.5lIj 3.5-4.0 rj 

Time spent on this course compared to other courses 



Percent of class attendance: 



9O-IOO5S 

75-89 
50-74 
25-49 
0-24 



of equal credit: considerably morel 

somewhat more 
about the same 
somewhat less 
considerably loss 



For the following ten quectTons, rate each 'item in two ways:*" " " 

a. give youi- actual rating of the instructor; then 

b. mcrk the second scale according to how you would prefer things to bco Of course, 
if you re satisfied, the second scale will bo marked the same as the first. 



From my observations, this instructor.., 



1. presents material in a clear & effective manner. 

(preferred) , o .. . 



or 

<o 



2 . ct imulGt c a int erect 

(prof orrod) « ^ . » ] 

3. makc.'3 work aocignmcntc & ctudont rooponoibilitiea clear. 

(preferred) =, * . o . . o 



<i o o a o • 



4- uces cxcvm itomo which stroos important aopecto of the course, 
(pref Grrod) . • . r , . c , ^ , , 



5- graden fairly and impart ially. 

(preferred) 



o a 0 • 0 o <j I 



6. demonotraten overall organization. . o . 

(preforrod) .« * o c 

7. mccto claon regularly and on timooo.. 

(proforred) , . . , , . . , 

0. fulfillc clafjjo objoctivec and obligat 
(preforrod) , 



lonc- o • 



9^ in acconrjible to otudentc both in cuid out of claon [ 

(preferred) „ ^ . ^ , ^ 



10„ CJhov/rj an intereot in -ind conooni for utudontn. 
(preferred) , 



00 



a PP 

3 on s:g 

O e+ fJO 

H- HO nc+ 



c 
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APPENDIX D (Form 4) 
ThG Univercity of Connecticut Survey of Courseo and Teaching 

Department ^ Couroe # Inatructor 

fijj'ect ions t Your inatructor will not ogg those ratings until after grades are 
awarded. Please be as objoctivo as you can, and DO NOT SI® YOUR ifiiME, 

Your semester;' / 2 5^ 

Expected grade in course: nil .1.1 u 

A B C D P Pass 

Percent of class attendance: 
90-1 00?5 
75-89 ■ 
50-74 ■ 
25-49 " 
0-24 



Is this course in your major field?! — ' □ 

yoB no 



1 .0-1 .5 
1 .5-2.0 
2.0-2.5 





2.5-3-0 






3.0-3.5 






3.5-4.0 





Time spent on this course compared to other courses 
of equal credit: considerably more 

somewhat more 
about the same' 
somewhat less^ 
considerably less 



quite a bit 



a great deal 



How mu ch ha ve you learned from this course? 

almost nothing a little a moderate 

amount 

How would you rate this instructor in general, overall teaching ability? 

poor less than average good excellent 

adequate 
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FIGURE 1 

Description of MANOVA's to Determine if Format Changes 
Alter Ratings Awarded 



Form 1 

Two global items on 




Form 2 

Two global items on 
bottom 




Form 4 

Two global items only > 
evaluative items absent 




< 




Form .1 

Ten evaluative items 
on bottom 




Form 2 

Ten evaluative items 
on top 




Form 3 

Ten evaluative items 
only, global absent 
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TABLE 1 

Summary of Three Principal Components Analyses of Ten Items 
Contained on Forms If 2, and 3 of the Teacher Rating Scale 





Form 1 


Form 2 




Form 3 


Item 


Loading 


Item 


Loading 


Item 


Loading 


2. 


.808 


Factor 
1. 


I 

.892 


1. 


.911 


1. 


.801 


2. 


.848 


2. 


.866 


8. 


.786 


8. 


.842 


6. 


.816 


6. 


.785 


6. 


,836 


8. 


.647 


10. 


.638 


10. 


.412 


10. 


.507 


3. 


.468 


3. 


.405 


5. 


.471 


5. 


.744 


Factor 
5. 


II 

.694 


3. 


.631 


4. 


.714 


7. 


.633 


7. 


.615 


3. 


.400 


4. 


.557 


4. 


.569 






3. 


.541 


9. 
10. 

5. 


.543 
.532 
.530 


9. 


.810 


Factor 
9. 


III 
.931 






7. 


.704 


10. 
5. 


.527 
.403 
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TABLE 2 

Means and Standard Deviations of the Pour Formats of the Teacher Hating Scale 



Form 1 Form 2 Form 3 Form 4 

N=120 _ N=1l8 N=125 N=121 

X Bd X Bd X sd X sd 



Item 



4.27 .80 4.37 . 88 4.26 .84 

4.22 .94 4.16 .95 4.11 .94 

3. 4.41 .93 4.39 .86 4.45 .87 

4. 4.38 1.10 4.02 1.23 4.15 .97 

5. 4.74 .49 4.60 .63 4.65 .69 

6. 4.26 .91 4.19 1.05 4.16 .92 

« 

7. 4.78 .55 4.82 .43 4.86 .37 

8' 4.47 .66 4.47 .82 4.43 .80 

9. 4.62 .66 4.58 .74 4.54 .65 

10- 4.63 .59 4.58 .77 4.57 .75 
Global 

1 4.04 .96 3.91 1.00 3.99 .84 

Global 

2 4.24 .76 4.27 .87 4.29 .83 
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